Response code for Chinese spiders? [closed]

Posted by pt2ph8 on Server Fault See other posts from Server Fault or by pt2ph8
Published on 2012-10-30T16:44:53Z Indexed on 2012/10/30 17:04 UTC
Read the original article Hit count: 160

Filed under:
|

My server is being "attacked" by Chinese spiders that don't respect the rules in my robots.txt. They are being very aggressive and using a lot of resources, so I'm going to set up some rules in nginx to block them by user agent.

Question: which response code should I return, 403, 444 (empty response in nginx) or something else? I'm wondering how the spiders will react to different status codes. What's the best practice?

© Server Fault or respective owner

Related posts about blocking

Related posts about spider